Magnification observation apparatus and method for photographing magnified image

ABSTRACT

A high tone image is saved in a versatile image file format to enhance usability. There are provided an imaging unit that images an original image having a predetermined dynamic range, which is a ratio between minimum luminance and maximum luminance; a synthesized image generating part that generates a synthesized image data that is higher in tone than a tone width of the original image by synthesizing a plurality of original images imaged under different imaging conditions at the same observation position; a display unit that displays the images imaged by the imaging unit; a tone conversion part that converts the synthesized image data generated by the synthesized image generating part to low tone image data having a tone width capable of being displayed on the display unit; and a tone data saving part that generates a high tone data-attached display file including a high tone image region for saving the synthesized image data serving as a basis as an image file for saving the low tone image data.

The present application claims priority from Japanese Patent ApplicationNo. 2007-146786, filed Jun. 1, 2007, which is incorporated herein byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a magnification observation apparatussuch as a digital microscope and a microscope that images and displays amagnified image, and a method for photographing a magnified image.

2. Description of the Related Art

A magnification observation apparatus that magnifies and displays asubject such as a sample including a minuscule object and a workpieceutilizes an optical microscope and digital microscope using an opticallens, and the like. The digital microscope receives reflected light ortransmitted light from an observation target fixed at an observationtarget fixing unit entering through an optical system with a lightreceiving device such as a CCD, which electrically reads the light forevery pixel arranged two-dimensionally, and displays the electricallyread image on a display unit such as a display (e.g., see JapaneseLaid-Open Patent Publication No. 2002-135648).

The sensitivity region of the light receiving device such as a CCD islimited to a particular range, whereas the sensitivity range of a humaneye is wider. Thus, the range that can be represented (dynamic range) ofthe image imaged with the CCD is limited compared to the human eye. Forinstance, the dynamic range is eight bits in a general JPEG image. Inthis case, if the tone of the image to be imaged exceeds such a range,saturation occurs, thereby causing underexposure, overexposure,halation, and the like. In order to solve such problems, used is a highdynamic range image (hereinafter referred to as “HDR image”) in which aplurality of low tone images are imaged with the dynamic range of theluminance region synthesized to obtain a high tone image. The HDR imageis obtained by synthesizing a plurality of images at different levels ofexposure of the same subject, and has a wide dynamic range from thedarkest shadow (black) to an extremely bright highlight (white). Forinstance, a plurality of eight bit images is synthesized to generate ahigh tone HDR image of sixteen bits or thirty-two bits, and such animage is saved. Overexposure occurs at the metal portion in the imageshown in FIG. 1, and underexposure occurs in the image shown in FIG. 2.When such images are synthesized, the HDR image as shown in FIG. 3 canbe generated. The portions of overexposure and underexposure in theoriginal image are clearly represented in the HDR image of FIG. 3.

When displaying the HDR image synthesized in the above manner on amonitor and the like, tone conversion (tone mapping) needs to beperformed to a color range that can be displayed on the monitor, thatis, a low dynamic range. Since, for example, only 16.77 million colorscan be represented in twenty-four bit color, and only 256 colors in thecase of eight bits in a general personal computer, the HDR image ofthirty-two bits is handled as twenty-four bits or eight bits throughtone mapping.

Various techniques have been proposed to widen the dynamic range usingthe HDR image. In order to resolve overexposure and underexposure bywidening the dynamic range, photographing is performed in the range fromdarkness to an extent where overexposure does not occur in any region ofthe image, that is, with a short exposure time, to brightness to anextent where underexposure does not occur in any region, that is, with along exposure time. Japanese Laid-Open Patent Publications Nos.2002-135648 and 2002-223387, for example, disclose techniques ofdetermining in which exposure time range a plurality of images bephotographed, using temporary photographing or the like, as a techniqueof controlling the exposure time in photographing an original image forsynthesizing an HDR image while changing the exposure time when wideningthe dynamic range.

Generally, such an HDR image is often used in applications to eliminatehalation contained in the image data, correct backlight, and the like.In the application of magnification observation, there may also be ademand to observe fine patterns contained in a narrow dynamic range asopposed to the application of literally widening the dynamic range forhalation measures and the like. In this case, tone images are finelyimaged and synthesized in a limited dynamic range to enhance the S/Nratio and luminance resolution. Therefore, the setting of exposure timesalso differs from that in the dynamic range widening application. Therehave not been examples of magnification observation apparatus usingchange in dynamic range for such an application. Also, there has notbeen a magnification observation apparatus capable of switching betweenthe dynamic range widening application and the luminance resolutionenhancing application, and automatically determining in whichapplication a user desires to photograph.

SUMMARY OF THE INVENTION

The present invention was made in view of such a background. It is amain object of the present invention to provide a magnificationobservation apparatus capable of automatically switching between adynamic range widening mode and a resolution enhancing mode according toobservation purposes, and a method for photographing a magnified image.

In order to achieve the above object, a magnification observationapparatus according to the present invention relates to a magnificationobservation apparatus capable of displaying an image obtained by imaginga sample to be imaged according to a set imaging condition, themagnification observation apparatus including an imaging conditionsetting part that sets at least an exposure time as the imagingcondition when imaging an original image in an imaging unit; the imagingunit that images the original image having a predetermined dynamicrange, which is a ratio between minimum luminance and maximum luminance,according to the imaging condition set in the imaging condition settingpart with respect to an observation position of the sample; asynthesized image generating part that generates synthesized image datathat is higher in tone than a tone width of the original image bysynthesizing a plurality of original images imaged under differentimaging conditions at the same observation position of the sample; adisplay unit that displays an image imaged by the imaging unit; and amode selecting part that selects, as a synthesized image photographymode for acquiring the synthesized image from the plurality of originalimages imaged in the imaging unit, either a dynamic range wideningphotography mode for generating the synthesized image having a dynamicrange wider than that of the original image, and a resolution enhancingphotography mode for enhancing a luminance resolution from the originalimage in a dynamic range narrower than that of the original image.According to such a configuration, a synthesized image in which theluminance resolution is enhanced within a narrow dynamic range can beacquired in addition to the dynamic range widening photography mode whengenerating the synthesized image, whereby the textures can be easilydistinguished even with respect to a sample that barely has contrastingdensity such as ceramic.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an image showing one example of an image with overexposure;

FIG. 2 is an image showing one example of an image with underexposure;

FIG. 3 is an image showing an HDR image in which the images of FIGS. 1and 2 are synthesized;

FIG. 4 is an outer appearance view showing a magnification observationapparatus according to an embodiment of the present invention;

FIG. 5 is a block diagram of a magnification observation apparatusaccording to a first embodiment of the present invention;

FIG. 6 is a block diagram of a magnification observation apparatusaccording to a second embodiment of the present invention;

FIG. 7 is a graph showing change in received light data with respect toheight z;

FIG. 8 is a block diagram showing a magnification observation apparatusthat generates a synthesized image;

FIG. 9 is a flowchart showing a procedure from photographing an HDRimage to displaying the HDR image on a display unit;

FIG. 10 is an image showing the ceramic surface;

FIG. 11 is an image showing a synthesized image in which FIG. 10 isimaged in a resolution enhancing photography mode;

FIG. 12 is a flowchart showing method 1 of automatically selecting asynthesized image photography mode;

FIG. 13 is a flowchart showing method 2 of automatically selecting thesynthesized image photography mode;

FIG. 14 is a flowchart showing method 3 of automatically selecting thesynthesized image photography mode;

FIGS. 15A to 15D are graphs showing exemplary histograms of luminancedistribution of a temporary image of sample A;

FIGS. 16A to 16D are graphs showing exemplary histograms of luminancedistribution of a temporary image of sample B;

FIG. 17 is an image showing one example of a user interface screen(simple mode) of a magnified image observation program;

FIG. 18 is an image showing one example of a user interface screen(detailed mode) of the magnified image observation program;

FIGS. 19A and 19B are images showing a state of adjusting a brightnessslider for a synthesized image imaged in a resolution enhancingapplication;

FIGS. 20A and 20B are images showing a state of adjusting a textureenhancement slider for the synthesized image imaged in the resolutionenhancing application;

FIGS. 21A and 21B are images showing a state of adjusting a contrastslider for a synthesized image imaged in a dynamic range wideningapplication;

FIGS. 22A and 22B are images showing a state of adjusting a color sliderfor the synthesized image imaged in the dynamic range wideningapplication;

FIG. 23 is a flowchart showing a procedure for tone-converting a hightone image to a low tone image;

FIG. 24 is a flowchart showing a setting procedure of a specific drawingparameter;

FIG. 25 is an image in one drawing parameter;

FIG. 26 is an image with which the drawing parameter is changed fromFIG. 25;

FIG. 27 is a schematic view showing a structure of a high tonedata-attached display file; and

FIG. 28 is a block diagram showing a flow of acquisition and display,and data saving of a synthesized image using a magnification observationapparatus according to the embodiments.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the present invention will now be described withreference to the drawings. It should be noted that the followingembodiments illustrate a magnification observation apparatus and amethod for creating a high tone image file for embodying the technicalidea of the present invention, and the present invention does not limitthe magnification observation apparatus and the method for creating ahigh tone image file creating to the following. The presentspecification never limits the members described in the appended Claimsto the members of the embodiments. In particular, dimensions, materials,shapes, relative arrangement, and the like of the components describedin the embodiments should not be taken as limiting the scope of thepresent invention unless specifically stated, but as merely exemplaryexamples. The size, position relationship, and the like of the membersshown in each figure are sometimes exaggerated to clarify thedescription. In the following description, the same names or referencenumerals indicate the same or equivalent members, and the detaileddescription will not be repeated. Furthermore, each componentconstituting the present invention may take a mode in which a pluralityof elements is configured with the same member and one member is alsoused as a plurality of elements, or the function of one member may beshared by a plurality of members.

The connection between a magnification observation apparatus used in theexamples of the present invention and a computer, a printer, an externalstorage device, and other peripheral equipment connected to theapparatus for operation, control, display, and other processes includesserial connection such as IEEE1394, RS-232x and RS-422, and USB;parallel connection; or electrical, magnetic or optical connectionthrough a network such as 10 BASE-T, 100BASE-TX, and 1000BASE-T toperform communication. The connection is not limited to a physicalconnection using wires and may be wireless LAN such as IEEE802.1x,wireless connection using electric wave, infrared or opticalcommunication such as Bluetooth. The recording medium for exchangingdata and saving the setting includes memory cards, magnetic discs,optical discs, magnetooptical discs, and semiconductor memories. In thepresent specification, the term “magnification observation apparatus” isnot limited to a magnification observation apparatus body and is used toinclude a magnification observation system combining peripheralequipment such as computers and external storage devices.

In the present specification, the magnification observation apparatusand the method for creating a high tone image file are not limited tothe system that generates an image file including a synthesized image,as well as an apparatus and a method for performing, in hardware,input/output, display, calculation, communication, and other processesrelating to the generation of an image file. The scope of the presentinvention also encompasses an apparatus and a method for realizing theprocesses in software. For instance, the apparatus and the system thatcause a general purpose circuit and a computer to incorporate softwareand programs, plug-ins, objects, libraries, applets, compilers, modules,macroinstructions on a specific program, and the like and perform imagegeneration or processes relating thereto also correspond to themagnification observation apparatus and the method for creating a hightone image file of the present invention. In the present specification,the computer includes, in addition to general purpose or dedicatedelectronic calculators, work stations, terminals, portable electronicequipment, mobile phones such as PDC, CDMA, W-CDMA, FOMA (registeredtrademark), GSM, IMT2000, and fourth generation, PHSs, PDAs, pagers,smartphones, and other electronic devices. In the present specification,the term “program” is not limited to one used alone and may be used in amode of functioning as part of a specific computer program, software, aservice, and the like, a mode of being invoked and functioning whennecessary, a mode of being provided as a service in an environment suchas an OS, a mode of being resident and operating in the environment, anda mode of operating in background, or as a support program.

First Embodiment

A magnification observation apparatus according to a first embodiment ofthe present invention will be described with reference to FIGS. 4 and 5.As shown in FIG. 4, the magnification observation apparatus includes anillumination unit 60 for illuminating a sample (subject) to be observed,an imaging unit 10 that images the sample illuminated by theillumination unit 60, and an information processing apparatus 50including a display unit 52 that displays a magnified image imaged withthe imaging unit 10. The magnification observation apparatus of FIG. 4also includes a sample fixing unit (stage 30 to be mounted with a sampleS) that fixes the sample, an imaging device (CCD 12) that electricallyreads reflected light or transmitted light from the sample S fixed tothe sample fixing unit entering through an optical system 11, and afocus adjusting unit (stage lifting/lowering unit 20) that adjusts afocus by changing the relative distance in an optical axis direction ofthe sample fixing unit and the optical system 11. As shown in FIG. 5,the information processing apparatus 50 includes a focal lengthinformation storage unit (memory 53) that stores focal lengthinformation relating to the relative distance in the optical axisdirection between the sample fixing unit and the optical system 11 whenthe focus is adjusted by the focus adjusting unit along withtwo-dimensional position information of the sample in a planesubstantially perpendicular to the optical axis direction, the displayunit 52 that displays an image read by the imaging device, a regionsetting unit (operation unit 55, pointing device 55A) capable of settingat least one of the regions of a portion of the image displayed by thedisplay unit 52, and a calculating unit (control unit 51) thatcalculates an average height in the optical axis direction of the sampleS corresponding to the region set by the region setting unit based onthe focal length information stored in the focal length informationstorage unit relating to a portion of or all of the sample Scorresponding to the region set by the region setting unit. Themagnification observation apparatus can calculate an average height(depth) in an optical axis direction of a sample corresponding to aspecified region using an imaging device that electrically readsreflected light or transmitted light from the sample fixed to the samplefixing unit entering through the optical system.

As shown in FIG. 5, the imaging unit 10 includes the stage 30, which isone form of the sample fixing unit to be mounted with the sample S; thestage lifting/lowering unit 20 that moves the stage 30; the CCD 12,which is one form of the imaging device that electrically reads thereflected light or the transmitted light entering through the opticalsystem to the sample fixed to the stage 30 for every pixel arrangedtwo-dimensionally; and a CCD control circuit 13 that drives and controlsthe CCD 12. Furthermore, the information processing apparatus 50 or themagnification observation apparatus body is connected to the imagingunit 10. The information processing apparatus 50 includes the memory 53,which is one form of an image data storage unit that stores image dataelectrically read by the imaging device; the display unit 52 such as adisplay or a monitor that displays an image based on the image dataelectrically read by the imaging device; the operation unit 55 thatperforms input and other operations based on the screen displayed on thedisplay unit 52; and the control unit 51 that performs image processingand various other processing based on the information inputted by theoperation unit 55. The display configuring the display unit 52 is amonitor capable of displaying images at high resolution and may be aCRT, a liquid crystal panel, or the like.

The operation unit 55 is connected to the computer by wire orwirelessly, or is fixed to the computer. The general operation unit 55includes various pointing devices such as a mouse, a keyboard, a slidepad, a pointing stick, a tablet, a joystick, a console, a rotaryselector, a digitizer, a light pen, a numerical keypad, a touchpad, andan acupoint. In addition to the operation of a magnification observationoperation program, the operation unit 55 can also be used to operate themagnification observation apparatus itself or the peripheral equipmentthereof. A touch screen or a touch panel may be used for the displayitself that displays an interface screen, so that the user can performinput or operation by directly touching the screen by hand, oralternatively or in addition thereto, voice input and other existinginput means may be used. In the example of FIG. 4, the operation unit 55is configured by a pointing device 55A such as a mouse 55 a.

FIG. 4 shows an outer appearance view of the magnification observationapparatus according to the embodiment of the present invention. A camera10 a including an optical system and an imaging device is attached to acamera attachment unit 43 fixed to a supporting column 42 extending in avertical direction from a stand board 41. The stage lifting/loweringunit 20 having the stage 30 to be mounted with the sample S attached toan upper portion is provided on the stand board 41. The camera 10 a andthe stage lifting/lowering unit 20 are connected to and controlled bythe information processing apparatus 50. The information processingapparatus 50 includes the display unit 52 and the operation unit 55 suchas the mouse 55 a. An observed image is displayed on the display unit52.

A computer 70 is connectable to the magnification observation apparatusor the information processing apparatus 50, where the magnificationobservation operation program can be separately installed in thecomputer 70 so that the magnification observation apparatus can beoperated from the computer 70 side. In the present specification, themagnification observation operation program for operating themagnification observation apparatus using the computer includes, inaddition to operation programs installed in a general purpose or adedicated computer externally connected to the magnification observationapparatus, an operation program incorporated in the informationprocessing apparatus 50 or the control unit of the magnificationobservation apparatus described above. The operation function or theoperation program for operating the magnification observation apparatusis incorporated in the magnification observation apparatus in advance.The operation program may be installed or updated to the magnificationobservation apparatus in the form of rewritable software, firmware, andthe like. Therefore, the computer that executes the magnificationobservation operation program includes the magnification observationapparatus itself in the present specification.

FIG. 5 shows a block diagram of the magnification observation apparatusaccording to the embodiment of the present invention. The informationprocessing apparatus 50 is configured by the display unit 52, the memory53 that stores control programs, focal length information, receivedlight data, two-dimensional information, and the like, an interface 54for the information processing apparatus 50 to communicate data with thecamera 10 a and the stage lifting/lowering unit 20, and the operationunit 55 for the operator to perform operations relating to themagnification observation apparatus. The stage lifting/lowering unit 20is configured by a stepping motor 21, a motor control circuit 22 thatcontrols rising/lowering of the stepping motor 21, and the like. Theimaging unit 10 includes a light receiving device such as the CCD 12 asthe imaging device, the CCD control circuit 13 that drives and controlsthe CCD 12, and the optical system 11 that images the transmitted lightor the reflected light of the light emitted on the sample S mounted onthe stage 30 from the illumination unit 60 onto the CCD 12.

[Pixel Offset Part]

The imaging unit 10 includes a pixel offset part that obtains highresolution greater than or equal to the resolution of the CCD 12 bypixel offset. The pixel offset achieves higher resolution bysynthesizing an image photographed with the subject shifted by half apixel pitch and an image before the shifting. A representative imageshifting mechanism includes a CCD drive method of moving the imagingdevice, an LPF inclining method of inclining the LPF, and a lens movingmethod of moving the lens. In FIG. 5, the apparatus includes an opticalpath shifting unit 14 that optically shifts an incident optical path ofthe reflected light or the transmitted light entered to the CCD 12through the optical system 11 from the sample S fixed to the stage 30 inat least one direction by a distance smaller than the interval of onepixel of the CCD 12 in the direction. The mechanism and the method forrealizing the pixel offset in one embodiment of the present inventionare not limited to the above configuration, and existing methods andmethods to be developed in the future can be appropriately used.

The information processing apparatus 50 changes the relative distance inthe optical axis direction, or the height in the z direction, betweenthe stage 30 serving as the sample fixing unit and the camera 10 aincluding the optical system 11 and the CCD 12 serving as the imagingdevice by inputting control data relating to the control of the steppingmotor 21 to the motor control circuit 22. Specifically, the informationprocessing apparatus 50 controls the rotation of the stepping motor 21by inputting control data necessary for the control of the stagelifting/lowering unit 20 to the motor control circuit 22, andraises/lowers the height z (position in the z direction) of the stage30. The stepping motor 21 generates a rotation signal corresponding tothe rotation. The information processing apparatus 50 stores the heightz of the stage 30 serving as information relating to the relativedistance in the optical axis direction between the sample fixing unitand the optical system 11 based on the rotation signal inputted throughthe motor control circuit 22. The stage serves as an observationpositioning part 30A that positions an observation position with respectto the sample. In the present embodiment, an example of changing therelative distance in the optical axis direction between the samplefixing unit and the optical system by changing the height of the stage30 has been described, but the height of the optical system 11 such asthe height of the camera 10 a may be changed with the stage 30 fixed.The stage may be provided on a head member, which is a member differentfrom the body, other than on the magnification observation apparatusbody, or the imaging unit that does not include the stage may beprovided on the head member. The imaging unit that does not include thestage can be attached to an attachment stand or can be carried by theuser. Such a head member is connected to the magnification observationapparatus body by a cable.

The CCD 12 can electrically read the amount of light received for everypixel arranged two-dimensionally in the x and y directions. The image ofthe sample S imaged on the CCD 12 is converted to an electrical signalaccording to the amount of light received in each pixel of the CCD 12,and further converted to digital data in the CCD control circuit 13. Theinformation processing apparatus 50 stores the digital data converted inthe CCD control circuit 13 as received light data D along witharrangement information (x, y) of pixel serving as two-dimensionalposition information of the sample in a plane (x, y direction in FIG. 5)substantially perpendicular to the optical axis direction (z directionin FIG. 5) in the memory 53. The plane substantially perpendicular tothe optical axis direction herein does not need to be strictly a planeforming 90° with respect to the optical axis, and merely needs to be anobservation plane within an inclination range to an extent where theshape of the sample can be recognized at the resolution of the opticalsystem and the imaging device.

An example of mounting the sample on the stage has been described as oneexample of the sample fixing unit in the above description, but, e.g.,an arm may be attached in place of the stage and the sample may be fixedat the distal end of the arm. Furthermore, the camera 10 a may hot onlybe attached to the camera attachment unit 43, but also be removablydisposed at a desired position and angle through a hand-held method andthe like.

The illumination unit 60 shown in FIG. 4 includes an epi-illumination60A for emitting epi-light to the sample, and a transmissionillumination 60B for emitting transmitted light. Such illuminations areconnected to the information processing apparatus 50 by way of anoptical fiber 61. The information processing apparatus 50 includes aconnector 62 that connects the optical fiber 61 and incorporates a lightsource (not shown) for giving off light to the optical fiber 61 throughthe connector 62. A halogen lamp and the like are used for the lightsource.

[Control Unit 51]

The control unit 51 serving as the control part controls to convert theimaged observed image to the resolution enabling display on the displayunit 52 and display the same. In the magnification observation apparatusof FIG. 4, the imaging unit 10 displays the observed image in which thesample S is imaged by the CCD 12 on the display unit 52. Generally, theperformance of the imaging device such as the CCD is higher than thedisplay capability at the display unit in many cases, and thus the imageis displayed in a reduced manner by lowering the resolution to a sizecapable of being displayed in one screen by, e.g., decimating the imageto display the imaged observed image on one screen. Assuming that theread resolution at the time of being read by the imaging unit 10 is afirst resolution, the image is displayed on the display unit 52 at asecond resolution lower than the first resolution.

Second Embodiment

A laser microscope will now be described with reference to FIG. 6 as amagnification observation apparatus according to a second embodiment ofthe present invention. In the magnification observation apparatus of thesecond embodiment, the camera serving as the imaging unit includes afirst imaging unit in which the reflected light of the light from afirst light source (laser 101) emitted on the sample S is received by afirst light receiving device (photodiode 112) through a first opticalsystem 100, and a second imaging unit in which the reflected light ofthe light from a second light source (white lamp 201) emitted on thesample S is received by a second light receiving device (CCD 212)through a second optical system 200.

The first imaging unit will be described first. The first optical system100 includes a laser 101 that emits monochromatic light (e.g., laserbeam) to the sample S, a first collimator lens 102, a polarization beamsplitter 103, a ¼ wavelength plate 104, a horizontal deflectionapparatus 105, a perpendicular deflection apparatus 106, a first relaylens 107, a second relay lens 108, an objective lens 109, an imaginglens 110, a pin hole plate 111, and a photodiode 112.

The semiconductor laser 101 that emits a red laser beam is used for thefirst light source, for example. The outgoing laser beam from the laser101, which is driven by a laser drive circuit 115, passes through thefirst collimator lens 102, and then passes the ¼ wavelength plate 104with the optical path changed by the polarization beam splitter 103.After being deflected in the horizontal (lateral) direction and theperpendicular (vertical) direction by the horizontal deflectionapparatus 105 and the perpendicular deflection apparatus 106, the laserbeam passes the first relay lens 107 and the second relay lens 108 to becollected on the surface of the sample S placed on the stage 30 by theobjective lens 109.

Each of the horizontal deflection apparatus 105 and the perpendiculardeflection apparatus 106 is configured by a galvanometer mirror and isprovided to scan the surface of the sample S with the laser beam bydeflecting the laser beam in the horizontal and perpendiculardirections. The stage 30 is driven in the z direction (optical axisdirection) by the stage lifting/lowering unit 20. The relative distancein the optical axis direction between the focus of the objective lens109 and the sample S thus can be changed.

The laser beam reflected by the sample S follows the optical path in theopposite direction. That is, the laser beam passes through the objectivelens 109, the second relay lens 108, and the first relay lens 107, andagain passes through the ¼ wavelength plate 104 after the horizontaldeflection apparatus 105 and the perpendicular deflection apparatus 106.As a result, the laser beam transmits through the polarization beamsplitter 103 and is collected by the imaging lens 110. The collectedlaser beam enters the photodiode 112 through the pin holes of the pinhole plate 111 provided at the focal position of the imaging lens 110.The photodiode 112 converts the amount of light received to anelectrical signal. The electrical signal corresponding to the amount oflight received is inputted to an A/D converter 113 via an outputamplifier and a gain control circuit (not shown) to be converted todigital data. An example of using a photodiode as the first lightreceiving device has been described herein, but a photo-multiplier andthe like may be used. The laser 101 is not limited to red laser, andblue or ultraviolet laser may also be used. The height data of highresolution is obtained by using such a short wavelength laser.

By use of the first imaging unit of such a configuration, height (depth)information of the sample S can be obtained. The principle thereof willbe briefly described below. When the stage 30 is driven in the zdirection (optical axis direction) by the stepping motor 21 and themotor control circuit 22 of the stage lifting/lowering unit 21 in theabove described manner, the relative distance in the optical axisdirection between the focus of the objective lens 109 and the sample Schanges. When the focus of the objective lens 109 is focused at thesurface (measurement target surface) of the sample S, the laser beamreflected at the surface of the sample S is collected by the imaginglens 110 through the optical path described above, and thus most of thelaser beam passes the pin holes of the pin hole plate 111. Therefore,the amount of light received by the photodiode 112 becomes the maximumat this point. On the other hand, when the focus of the objective lens109 is shifted from the surface (measurement target surface) of thesample S, the laser beam collected by the imaging lens 110 is focused ata position shifted from the pin hole plate 111, and thus only part ofthe laser beam can pass through the pin holes. As a result, the amountof light received by the photodiode 112 thus significantly decreases.

Therefore, the height of the stage 30 where the amount of light receivedbecomes the maximum can be obtained by detecting the amount of lightreceived by the photodiode 112 at any point on the surface of the sampleS while driving the stage 30 in the z direction (optical axisdirection).

Actually, the amount of light received by the photodiode 112 is acquiredby scanning the surface of the sample S by the horizontal deflectionapparatus 105 and the perpendicular deflection apparatus 106 for everystep movement of the stage 30. FIG. 7 shows change in received lightdata D with respect to the height z of the stage 30 at one point(pixel). When the stage 30 is moved in the z direction from the lowerend to the upper end of the measurement range, the received light data Dthat changes according to the height z is obtained as shown in FIG. 7for a plurality of points (pixels) within the scanning range. Themaximum amount of light received and the focal length Zf thereat areobtained for each point (pixel) based on the received light data D. Theheight of the stage 30 corresponding to the maximum value of thereceived light data D becomes the focal length Zf. Therefore,distribution in the x-y plane of the surface height of the sample S isobtained based on the focal length Zf. This process is performed by thecontrol unit 51 based on the received light data D of the CCD 12 and thearrangement information (x, y) and the height information z of the pixelinputted through the interface 54 and stored in the memory 53.

The distribution of the obtained surface height is displayed on thedisplay unit 52 through various methods. For instance, the heightdistribution (surface shape) of the sample may be three-dimensionallydisplayed by three-dimensional display. Alternatively, the height datamay be converted to luminance data to be displayed as two-dimensionaldistribution of brightness. The height data may be converted to colordifference data so that distribution of height is displayed asdistribution of color.

In the second embodiment as well, the region is set to a rectangularshape by specifying two points on the image on the display unit 52 bymeans of the pointing device 55A and the like based on the height dataobtained by the first imaging unit, the average height in the region andthe relative height between each region are calculated and displayed onthe display unit 52, as in the first embodiment.

The surface image (black and white image) of a sample w is obtained froma luminance signal in which the amount of light received obtained foreach point (pixel) within the x-y scanning range is assumed as luminancedata. If the luminance signal is generated with the maximum amount oflight received at each pixel as luminance data, there can be obtained aconfocal image having a very deep depth of field focused at each pointat different surface height. If fixed at a height (position in the zdirection) at which the maximum amount of light received is obtained ata certain pixel of interest, the amount of light received at the pixelof a portion having a large difference in height from the portion ofpixel of interest becomes significantly small, and thus there isobtained an image in which only the portion at the same height as thatof the pixel of interest are bright.

The second imaging unit will now be described. The second optical system200 includes a second light source 201 for emitting white light(illumination light for color image photographing) to the sample S, asecond collimator lens 202, a first half mirror 203, a second halfmirror 204, and a CCD 212 serving as a second light receiving device.The second optical system 200 commonly uses the objective lens 109 ofthe first optical system 100, and the optical axes of the opticalsystems 100 and 200 are coincident.

A white lamp or the like is used for the second light source 201, butnatural light or indoor light may be used without providing a dedicatedlight source. The white light emitted from the second light source 201passes through the second collimator lens 202 and is then collected atthe surface of the sample S placed on the stage 30 by the objective lens109 with the optical path bent by the first half mirror 203.

The white light reflected by the sample S passes through the objectivelens 109, the first half mirror 203, and the second relay lens 108, andis reflected by the second half mirror 204 to be entered into the CCD212 capable of receiving light in color and then imaged. The CCD 212 isprovided conjugate with or at a position close to being conjugate withthe pin holes of the pin hole plate 111 of the first optical system 100.The color image imaged by the CCD 212 is read by the CCD control circuit213 and converted to digital data. The color image obtained in thismanner is displayed on the display unit 52 as a magnified color imagefor observation of the sample S.

The confocal image having a deep depth of field obtained in the firstimaging unit and the normal color image obtained in the second imagingunit may be combined to generate a color confocal image having a deepdepth of field focused at all the pixels, and then displayed. The colorconfocal image can be easily generated by, e.g., replacing the luminancesignal configuring the color image obtained in the second imaging unitwith the luminance signal of the confocal image obtained in the firstoptical system 100.

The magnification observation apparatus equipped with the first imagingunit including the first optical system 100 or a confocal optical systemand the second imaging unit including the second optical system 200 or anon-confocal optical system has been described, but the configurationmay be one including only the first imaging unit.

If the light receiving device is a two-dimensional imaging device (e.g.,CCD) that reads the amount of light received for every pixel arrangedtwo-dimensionally, and the focus adjusting unit has a configuration ofadjusting the focus based on the sum of the amount of light receivedcorresponding to part of or all of the sample corresponding to theregion set by the region setting unit, as in the magnificationobservation apparatus according to the first embodiment, the height ofthe sample can be measured with a simple configuration without requiringa complex configuration such as a confocal optical system. Inparticular, in such a magnification observation apparatus, the maximumvalue of the received light data is determined from the change in thereceived light data with respect to the relative distance on the basisof region set by the operator, that is, considerable number of pixels,and not on the basis of pixel, and the average height is calculatedbased on the average focal length at that point in time, and thusvariation in change with respect to the focal length of the receivedlight data at each pixel can be reduced and highly reliable measurementof average height can be performed even when the CCD is used as thelight receiving device with white light as the light source.Furthermore, when using the color CCD as the two-dimensional imagingdevice, the received light data of the pixel may be calculated based onthe received light data of RGB, or the received light data of the pixelmay be obtained based on the received light data of one or two colortones of RGB.

If the region set by the region setting unit is larger than the size ofthe sample and thus contains the entire sample, the portion other thanthe sample, that is, the upper surface of the stage is preferablyexcluded from the target of average height calculation. This is becausea more accurate sample height can be calculated. In this case, the uppersurface of the stage can be determined from, e.g., whether or not thedifference in height between a pixel and a pixel adjacent to the pixelis greater than or equal to a predetermined height. Obviously, even ifthe region set by the region setting unit is part of the sample, it ispreferably excluded from the target of average height calculation if theupper surface of the stage is in the region.

In the above embodiments, an example of electrically reading thereflected light from the sample fixed to the sample fixing unit has beendescribed, but light may be emitted from the rear surface of the sampleand the transmitted light may be electrically read.

(High tone Image)

FIG. 8 shows a block diagram of the magnification observation apparatusthat images a plurality of original images, synthesizes the images andgenerates a high tone synthesized image. The magnification observationapparatus shown in the figure includes a body 50A configuring themagnification observation apparatus body and the imaging unit 10. Thebody 50A and the imaging unit 10 are connected with a cable. In theexample of FIG. 8, the cable is configured by an optical fiber 61 forgiving off illumination light from the illumination light source 201A,and a signal line 63 for transmitting/receiving data between the body50A and the imaging unit 10. The body 50A transmits an imaging unitcontrol signal for controlling the imaging unit 10 to the imaging unit10 via the signal line 63, and the imaging unit 10 transmits an imagedimage signal to the body 50A. The signal line for the imaging unitcontrol signal and the signal line for the image signal may be providedindividually.

The imaging unit 10 includes an imaging device 12A such as a CCD and aCMOS and the illumination unit 60, where the illumination light isemitted from the illumination unit 60 to the sample S, and the reflectedlight thereof is imaged with the imaging device 12A. The body 50Aincludes the imaging control unit 13A that controls the imaging unit 10,an illumination light source 201A that generates the illumination light,a body control unit 51A connected to the imaging control unit 13A andthe illumination light source 201A, and the display unit 52 connected toa body control unit 51A for displaying images and necessary information.The body control unit 51A transmits control signals to the imagingcontrol unit 13A and the illumination light source 201A to control theoperations thereof The body control unit 51A also includes an imagecalculating unit 81 that retrieves image data received by the imagecontrol unit 13A from the imaging unit 10 and performs processes such assynthesis, the memory 53 that holds image data and various set values, amode selecting part 82 for selecting a synthesized image photographymode to be described later, and an imaging condition setting part 83that sets the imaging conditions in the imaging unit 10. The imagecalculating unit 81 functions as a synthesized image generating part 85that synthesizes a plurality of original images and generates high tonesynthesized image data, a tone conversion part 86 that performs toneconversion, and a drawing setting part 87 that sets drawing parametersfor the tone conversion part 86 to tone-convert a high tone image to lowtone image data. Such a body control unit 51A can be configured by anASIC, an LSI, and the like.

The image data and the set content held in the memory 53 can bedisplayed on the display unit 52 connected to the body control unit 51A.A monitor such as a CRT, a liquid crystal display, an organic EL, or thelike may be used for the display unit 52. The operation unit 55 for theuser to perform various operations on the body control unit 51A isconnected to the body 50A. The operation unit 55 is an input device suchas a console or a mouse. In this example as well, the display unit andthe operation unit can be integrally incorporated in the body or may beexternal members. If the display unit is configured by a touch panel,the display unit and the operation unit can be integrally configured.

(Synthesized Image Photography Mode)

The magnification observation apparatus includes a dynamic rangewidening photography mode suitable for a dynamic range wideningapplication and a resolution enhancing photography mode of enhancingluminance resolution and enhancing contrast as synthesized imagephotography modes for acquiring the synthesized image in the synthesizedimage generating part 85. Generated in the dynamic range wideningphotography mode is a synthesized image having a wider dynamic rangethan the original image. Generated in the resolution enhancingphotography mode is a synthesized image in which the luminanceresolution is enhanced from the original image in a dynamic rangenarrower than the dynamic range of the imaging device.

(Dynamic Range Widening Photography Mode)

A so-called HDRI is imaged in the dynamic range widening photographymode. The HDRI (High Dynamic Range Image, hereinafter referred to as“HDR image”) is an image in which the dynamic range, that is, the ratiobetween the minimum light amount and the maximum light amount issignificantly higher than conventional images. For instance, on themonitor of a standard computer, the color of eight bits to twenty-fourbits is adopted as a standard color representation so as to berepresented in 256 to 16.77 million tones, but more colors exist inreality, and the eyes of a human view an image by adjusting the image toa reference brightness that appears to be appropriate by changing thesize of the pupil. The HDR image with a greater amount of colorinformation that exceeds the representation capability and the like ofthe monitor is therefore used. An existing method such as synthesizing aplurality of images obtained by imaging the same observation target atthe same position and under different imaging conditions (typically,exposure time of the imaging device) may be used to acquire such an HDRimage.

However, the colors of greater than or equal to eight bits totwenty-four bits (256 to 16.77 million colors) cannot be represented onthe monitor, where overexposure occurs in colors brighter than the colorrange that can be represented and underexposure occurs in colors darkerthan the color range. Since the texture and the like of the imagesometimes cannot be recognized in this state, tone conversion (tonemapping) is appropriately performed to convert the image to a low toneimage (hereinafter also referred to as Low Dynamic Range Image (LDRimage)) in which portions of overexposure and underexposure can berecognized, when the HDR image is displayed on the monitor and the like.For instance, overexposure occurs at the metal portion in the imageshown in FIG. 1, but that portion is clearly seen in the image shown inFIG. 3. This is because the image of FIG. 3 has data of a portion havinga light amount greater than the light amount that can be photographed inFIG. 1. In a general environment, the light amount data has a dynamicrange of about 100000:1 at a maximum.

Various methods have been proposed for creating the HDR image, but inthe present embodiment, the HDR image is created by synthesizing aplurality of original image groups obtained by photographing a sample atthe same position while changing the exposure time or the shutter speedof the imaging device herein, in the above-described manner. Forinstance, the HDR image shown in FIG. 3 can be created by synthesizingthe dark image of FIG. 2 and the bright image of FIG. 1.

Since the dynamic range of the HDR image may be 100000:1, the HDR imagecannot be represented with the usual eight-bit image of 256 tones. Thus,the data is generally represented in floating point. Thirty-two bitsingle precision floating point, sixteen bit floating point, or the likeis used for the format for saving the file. When drawing the HDR image,the HDR image cannot be displayed as it is on a usual monitor of 256tones since the dynamic range is wide. If the HDR image is displayed bysimply converting (mapping) the image to 256 tones, the dark portionsometimes cannot be seen or the bright portion sometimes becomes toobright. Thus, the image needs to be adjusted so that the dark portionand the bright portion can be moderately viewed before displaying thesame. Specifically, the image is tone-converted to a tone mapping imageon which image processing is performed so as to greatly squeeze thebright portion and raise the dark portion. Thus, fine textures, whichare difficult to be seen in the normal tone mapping image, can bedisplayed in an enhanced manner. The procedure from photographing an HDRimage to displaying the same on the display unit as described above willbe described with reference to the flowchart of FIG. 9.

First, in step S901, a plurality of original images is photographedunder different imaging conditions. An original image of eight bit isimaged in the imaging unit 10. In this case, the exposure time isappropriately controlled so that at least one image free of overexposureand underexposure can be taken in any region. The number of imaging isappropriately determined according to the image quality, the wideness ofthe necessary dynamic range, and the like of the HDR image. In stepS902, the HDR image is synthesized. The HDR image is synthesized in thesynthesized image generating part using the eight-bit image groupphotographed in step S901. In step S903, the HDR image istone-converted. The synthesized HDR image is tone-mapped in the toneconversion part to create a tone mapping image converted to a tone widthcapable of being displayed on a monitor and the like. In this example,conversion is made to 256 tones. In this case, the texture enhancementprocess and the like are also appropriately performed.

Finally, in step S904, the obtained HDR image and the tone mapping imageare saved. The HDR image is thirty-two bit floating point data, wherethe HDR image data is converted to an appropriate file format and savedwhen saving the file. The tone mapping image is saved in a versatileimage format such as JPEG and TIFF, and the HDR image or original datafrom which the tone mapping image is generated is saved as metadata.Furthermore, the parameters used at the time of creating the tonemapping image are also saved. Thus, the user can display the tonemapping image with a general purpose image display program and canreplace the tone mapping image by using a dedicated image displayprogram. That is, another tone mapping image may be generated from theHDR image by adjusting the parameters, and such a tone mapping image canbe newly overwritten on the metadata of the HDR image and saved alongwith the parameters at the time of conversion. The user can not onlybrowse the tone mapping image but can also change the tone mapping imagecontained in the file to a desired tone mapping image in the abovemanner. The HDR image is used in, e.g., applications where saturationsuch as overexposure and underexposure be suppressed in the image.

The HDR image is obtained by synthesizing images of dynamic ranges widerthan the dynamic range of the imaging device, but when the dynamic rangeof the imaging device is sufficiently wide, that is, when theperformance of the imaging device itself is enhanced and a sufficientdynamic range can be covered with one imaging, such one original imagecan be handled in a similar manner to the synthesized image. In thiscase, the HDR image is acquired only with the imaging device, and thusthe synthesized image generating part becomes unnecessary.

(Resolution Enhancing Photography Mode)

Opposite to the dynamic range widening photography described above,photographing in which the resolution is enhanced so that fine patternscan be displayed in a narrow dynamic range is also possible.

In the resolution enhancing photography mode, the images of whichimaging conditions are finely changed are synthesized in a dynamic rangenarrower than the original image to obtain a synthesized image whichluminance resolution is enhanced from the original image. Thesynthesized image obtained here is literally not an HDR image since thedynamic range is not widened, but is a high tone image similar to theHDR image that can be included in the HDR image in the presentspecification for the sake of convenience. Furthermore, in the presentspecification, the HDR image is used to mean that the dynamic range iswider than the dynamic range capable of being displayed on the displayunit, but is not limited thereto, and may refer to an image of which thedynamic range is wider than the dynamic range capable of being imaged bythe imaging device of the imaging unit or an image having a specificnumber of bits that is greater than or equal to, e.g., twenty-four bitsor thirty-two bits.

Generally, the HDR image is often applied to applications where halationcontained in the image data is eliminated or backlight is corrected, andthe like using the wide dynamic range. The HDR image also enables finetextures, which were barely seen, to be seen by retrieving signalsembedded in noise/quantization errors through synthesis of a pluralityof images with respect to a sample that barely has contrasting density,and performing the texture enhancement process. In this case, a hightone image in which the S/N ratio and the luminance resolution areenhanced is obtained by finely imaging tone images and synthesizing thesame in a limited dynamic range. As one example, an image of a ceramicsurface is shown in FIG. 10. The pattern is barely recognized at theceramic surface in this state, but through resolution enhancingphotography, the pattern of the surface can be enhanced and thetextures, which were barely seen, can be distinguished as shown in FIG.11.

The HDR technique has been developed taking use in the dynamic rangewidening application into consideration and has not been used inresolution enhancing photography application of opposite idea. Thus, theapplication can be extended and use in a wider variety of applicationsbecomes available by achieving a magnification observation apparatusthat can be switched between widening of dynamic range and enhancing ofresolution according to the observation applications.

Therefore, the resolution enhancing photography mode is used in, e.g.,applications where fine patterns and contrast in the image are desirablyenhanced. In the resolution enhancing photography mode, the amount ofchange in exposure time is set smaller than that set in the dynamicrange widening photography mode.

(Mode selecting Part 82)

Either one of the synthesized image photography modes is selected by themode selecting part 82. The mode selecting part 82 automaticallydetermines an appropriate synthesized image photography mode based onimage analysis. Alternatively, the synthesized image photography modemay be manually selected by the user. For instance, the user can operatethe operation unit 55 to select a desired synthesized image photographymode. The selected synthesized image photography mode may be clearlyshown on the display unit, thereby informing the user with whichsynthesized image photography mode the photography is being performed.

(Automatic Selection by the Mode Selecting Part)

A method in which the mode selecting part automatically selects thesynthesized image photography mode will now be described. Three methods1 to 3 will be described with reference to the flowcharts of FIGS. 12 to14, respectively.

(Method 1: Analysis of Observed Image Observed before Photographing)

First, a method of analyzing an observed image observed beforephotographing will be described with reference to the flowchart of FIG.12. In this method, the presence of a saturated region is searched fromthe image being displayed on the display unit, and the exposure time tobe used in the photographing is determined. When photographing in thedynamic range widening application, a saturated region such asoverexposure or underexposure is assumed to exist at all times in aportion of the observed image which the user is observing in the currentsetting. When enhancing the contrasting density, the object beingobserved is most likely to be an object that barely has contrastingdensity. Thus, the contrast of the observed image is assumed to be smalland saturation is not assumed to occur. Thus, the presence of asaturated region is searched from the image being displayed on thedisplay unit, and the exposure time used in the photographing thereof isdetermined.

To describe the specific procedure, first, in step S1201, the imageobserved before photographing of the original image is acquired. At thepoint where acquisition of synthesized image is instructed, the observedimage that has been displayed on the display unit is invoked. In stepS1202, the mode selecting part determines whether or not the observedimage contains a saturating portion. If the observed image contains asaturating portion, the process proceeds to step S1203 and the dynamicrange widening photography mode is set. If the observed image does notcontain a saturating portion, the process proceeds to step S1204 and theresolution enhancing photography mode is set. This method allowsdetermining and selecting the appropriate synthesized image photographymode in the simplest manner. The imaging conditions corresponding toeach selected synthesized image photography mode can be automaticallyset as necessary. Thus, the user can obtain the synthesized imagewithout being conscious of the type of synthesized image photographymode, and there is provided an advantage that even users who are notfamiliar with synthesized image can perform the operation.

(Method 2; Analysis of Temporarily Photographed Plurality of TemporaryImages)

The method of analyzing a plurality of temporarily photographedtemporary images will be described with reference to the flowchart ofFIG. 13 and the histograms of FIGS. 15 and 16.

(Temporary Synthesis)

In this method, prior to the imaging of original image, a plurality oftemporary images is first photographed under temporary imagingconditions set in advance, and then determination is made on whether ornot there are any temporary image in which a saturated region does notexist for each temporary image in the mode selecting part. The temporaryimaging conditions are conditions for easily acquiring an image, where aplurality of temporary images is imaged while appropriately changing theconditions since the tendency of the images merely needs to be grasped.Thus, the setting may be rougher than the imaging conditions of theoriginal image necessary for the generation of synthesized image, thenumber of imaging of the temporary image is set less than the number oforiginal images necessary for the generation of synthesized image, andthe changing width of the exposure time is also set to be large. In thisexample, the temporary synthesized image is synthesized with thesynthesized image generating part, but a calculation part for easilysynthesizing the temporary images at high speed may be separatelyprepared.

FIGS. 15A to 15D and 16A to 16D show exemplary histograms of theluminance distribution of an imaged temporary image. In this example,four images are taken for the same site in different samples A and Bwhile roughly changing temporary imaging conditions, where FIGS. 15A to15D show a sample A and FIGS. 16A to 16D show a sample B. Such temporaryimages are displayed in the order of changing the exposure time as atemporary imaging condition. That is, the exposure time is changed so asto become longer in the order of FIGS. 15A to 15D or FIGS. 16A to 16D.

Regarding the sample A, a saturated region exists in all of FIGS. 15A to15D. The fact that a saturated region exists in all the temporary imagesindicates that the dynamic range for grasping the entire luminanceinformation contained in the sample is not sufficient. Therefore,determination is made that the dynamic range widening photography modeis suited for this sample in the mode selecting part.

Regarding the sample B, a saturated region is found in FIGS. 16A and16D, but a saturated region is not found in FIGS. 16B and 16C. Thus, itis apparent that all the luminance distribution can be made fine withone temporary image for the sample B, that is, the dynamic range of theluminance distribution is narrow. In this case, it is preferable toacquire detailed information by finely changing the exposure time withinthe narrow dynamic range. Therefore, determination is made that theresolution enhancing photography mode is suited for the sample B.

The specific procedure of the above will be described with reference tothe flowchart of FIG. 13. First, in step S1301, temporary imaging isperformed a plurality of times while changing the shutter speed. In stepS1302, determination is made on the plurality of temporary imagesobtained in this manner as to whether a saturated region exists in everytemporary image by the mode selecting part. If a saturated region existsin all the temporary images, the process proceeds to step S1303, and thedynamic range widening photography mode is selected. If a saturatedregion does not exist, i.e. if there is a temporary image that does notcontain a saturated region, the process proceeds to step S1304, and theresolution enhancing photography mode is selected. In the examples ofFIGS. 15A to 15D and 16A to 1D, determination is made by the modeselecting part that FIGS. 15A to 15D are suited for the dynamic rangewidening photography mode, and FIGS. 16A to 16D are suited for theresolution enhancing photography mode.

As described above, an appropriate exposure time can be automaticallyset for the imaging condition corresponding to each selected synthesizedimage photography mode, as necessary. In particular, in this method, arange for changing the exposure time to be set in each synthesized imagephotography mode can be simultaneously obtained in addition to automaticselection of the appropriate synthesized image photography mode. Thatis, in the case of the dynamic range widening photography mode, theexposure time is changed in a range from the exposure dark enough sothat overexposure is eliminated to the exposure bright enough so thatunderexposure is eliminated. In the resolution enhancing photographymode, and the exposure time is set such that saturation does not occur,the exposure time being for the portion at which the difference incontrasting density appears the most. Thus, not only the setting of thesynthesized image photography mode, but the setting of the imagingcondition in each synthesized image photography mode can be automated.

(Automatic Setting of the Exposure Time by Method 2)

Details for setting the range in which the exposure time is to bechanged as the imaging condition in each synthesized image photographymode will be first described for the dynamic range widening photographymode. In a histogram, when the luminance is imaged at a resolution ofeight bits, a saturated region is assumed to exist if either the left orthe right end has a value in the luminance distribution range of 0 to255. For instance, the luminance has a value (luminance 0) on the axison the left side in FIG. 15A, and thus a region darker than the aboveexists, whereby the fact that the data is not detected, that is,occurrence of underexposure is recognized. The vicinity of the axis onthe right side (luminance 255) does not have a luminance value. That is,it is found that overexposure is not occurring. Furthermore,determination can be made that the upper limit of the brightestluminance is the circled position in FIG. 15A.

In FIG. 15D, the luminance value is shown on the axis on the right sideand occurrence of overexposure can be recognized, but the luminancevalue is not indicated in the vicinity of the axis on the left side.This means that the luminance information of the darkest region issufficiently complemented in FIG. 15D, and the circled region of FIG.15D is determined as the lower limit of the darkest luminance.

Since the upper limit and the lower limit of the range of the luminancedistribution can be detected with respect to the sample A, the exposuretime is set so that luminance information can be accurately detectedwithin the range. That is, the exposure times corresponding to theluminance values of the upper limit and the lower limit are obtainedfrom a calculation formula, a lookup table, and the like, and theexposure time is changed within the range, so that a high tone image canbe efficiently imaged without acquiring useless luminance information.

A method of determining the range for changing the exposure time in theresolution enhancing photography mode will now be described withreference to FIGS. 16A to 16D. In the example of FIGS. 16A to 16D, asaturated region exists in the histograms shown in FIGS. 16A and 16D andthus the remaining histograms shown in FIGS. 16B and 16C are extractedas described above. A temporary image in which difference in contrastingdensity is represented in the widest range is selected from thetemporary images. The distribution in the histogram is represented widerin FIG. 16B than in FIG. 16C. Thus, the luminance information can beefficiently acquired without wasting the acquirable range of luminancedistribution using the exposure time adopted in the histogram shown inFIG. 16B. An appropriate exposure time is thereby selected from thetemporary imaging conditions used when the temporary images areacquired.

(Method 3 Method by Temporary Imaging and Analysis of TemporarySynthesized Image)

Lastly, a method by temporary imaging and analysis of the temporarysynthesized image will now be described with reference to the flowchartof FIG. 14. Similar to the above, in step S1401, a plurality oftemporary images is imaged under a temporary imaging condition set inadvance. The exposure time is greatly changed as compared with, e.g.,the imaging of the original image, and the number of images issuppressed to a relatively small number. In step S1402, the plurality oftemporary images is synthesized to generate a temporary synthesizedimage. In step S1403, the dynamic range, that is, the ratio between theminimum luminance and the maximum luminance, of the temporarysynthesized image is calculated, and determination is made whether ornot such a dynamic range is greater than the dynamic range of theimaging device of the imaging unit 10. If the dynamic range of thetemporary synthesized image is greater than the dynamic range of theimaging unit 10, the process proceeds to step S1404, and the dynamicrange widening photography mode is selected since the dynamic range iscurrently insufficient. If the dynamic range of the temporarysynthesized image is smaller, the process proceeds to step S1405, andthe resolution enhancing photography mode in which the luminanceresolution is enhanced is selected since the dynamic range of theimaging unit 10 is sufficient.

(Automatic Setting of Exposure Time by Method 3)

In this method as well, an optimum exposure time in each synthesizedimage photography mode can be obtained. The following relationalexpression is established between the luminance I of the image, thelight amount L of the scene, the exposure time t, and the responsefunction of the camera (function representing what value the pixel valuetakes with respect to how much light amount enters the imaging deviceand how long exposure is performed) F.I=F(L*t)

F is measured and acquired in advance. The exposure time t₁ to t₂ areset to the following range so that the minimum/maximum light amountL_(min) and L_(max) of the temporary synthesized image becomes the pixelvalue I_(min) (minimum pixel value at which underexposure does notoccur, e.g. about 30) and I_(max) (maximum pixel value at whichoverexposure does not occur, e.g. about 220), respectively, in the caseof the dynamic range widening photography mode.t ₁ =F ⁻¹(I _(min))/L _(min)t ₂ =F ⁻¹(I _(max))/L _(max)

In the case of the resolution enhancing photography mode, the exposuretime of one of the following (1) and (2) is set in the range (t₁ to t₂)in which the minimum/maximum light amounts do not saturate.

(1) exposure time in which the difference between I_(min) and I_(max)becomes the largest

(2) exposure time satisfying L₁*t=x_max by calculating the averagevalue, median value L₁, and the like of the light amount L of the scene

where x_max is a value of x at which the slope (derivative value F′) ofF(x) becomes a maximum.

(Frame Average Value)

When generating a synthesized image in the synthesized image generatingpart in the resolution enhancing photography mode, in addition to themethod of imaging the original image while varying the exposure time, itis possible to adopt a method of imaging a plurality of original imagesin the imaging unit with the exposure time fixed to a constant value andtaking a frame average value to generate synthesized image data. Thatis, noise is generally mixed in digital images, and such noise is addedat an average 0 according to the normal distribution. Thus, theresolution caused by quantization errors can be enhanced whileeliminating the influence of noise by taking the average value. Forinstance, with respect to a pixel whose true value is 100.1, 100 isoutputted due to a quantization error if noise is not present, butvariation of 98, 104, 101, 97, and the like occurs for every imagingsince noise is present. A value after the decimal point (100.1) can beobtained while eliminating the influence of noise by taking the average.According to such a configuration, a synthesized image of highresolution can be generated with the exposure time fixed.

Comparing the above methods, the analysis of the image observed beforephotographing of method 1 is the most convenient method, and the amountof calculation increases in the order of the method by analysis oftemporarily photographed plurality of temporary images of method 2, andthe method by temporary imaging and analysis of temporary synthesizedimage of method 3. On the other hand, the calculation up to the optimumexposure time in each application can be carried out by the methods of 2and 3.

(Weighted Imaging Condition)

Furthermore, it is possible to adopt, not only a configuration ofalternatively selecting either the dynamic range widening photographymode or the resolution enhancing photography mode in the mode selectingpart, but a configuration of setting an imaging condition obtained byweighing and combining the imaging conditions for both the photographymodes. For instance, the photograph application is analyzed in the modeselecting part, a weighting coefficient for each of the dynamic rangewidening photography mode and the resolution enhancing photography modeis calculated, and a weighted imaging condition is accordingly set bythe imaging condition setting part. Thus, a weighted imaging conditioncombined in view of the balance between the two modes can be set withoutfixing the photograph application to either the dynamic range wideningphotography mode or the resolution enhancing photography mode, and thusmore flexible imaging can be performed. In calculating the weighting inthe mode selecting part, the ratio between both the photography purposesis determined, and a weighted imaging condition corresponding thereto isset. Specifically, an appropriate imaging condition is set according tohow much the weight of halation removal is, or whether or not muchweight is placed on the enhancement of contrast. For instance, indetermining whether or not a saturated portion is contained in theobserved image observed before photographing in step S1202 of FIG. 12,the number and the area of the saturated portion are checked and theweight of each photography mode is determined by the mode selecting partaccording thereto, instead of uniformly setting the dynamic rangewidening photography mode when a saturated portion is contained and theresolution enhancing photography mode when a saturated portion is notcontained. As one example, the weight of dynamic range wideningphotography mode is increased the greater the number of saturatedportion or the larger the area ratio. Similarly, instead of determiningwhether a saturated region exists in all the plurality of temporaryimages in step S1302 of FIG. 13, the mode selecting part may set theweighted imaging condition to give weight to the dynamic range wideningphotography mode the greater the number of temporary images including asaturated region is and to the resolution enhancing photography mode thegreater the number of temporary images not including a saturated regionis. Furthermore, in determining whether the dynamic range of thetemporary synthesized image is greater than the dynamic range of theimaging device in step S1403 of FIG. 14, the extent of largeness isdetermined, and the mode selecting part sets the weighted imagingcondition to give weight to the dynamic range widening photography modethe greater the ratio or the difference is, and to the resolutionenhancing photography mode the smaller the ratio or the difference is.Thus, the criterion is not alternative in the process of determining thephotography mode, and more appropriate imaging is realized by settingthe imaging condition so as to continuously change the weight.

(Exposure Time Adjustment Part 84)

There may be provided the exposure time adjustment part 84 for furtheradjusting the exposure time for the original image set by the imagingcondition setting part 83 to a desired value. The user adjusts theimaging condition such as exposure time to a desired value from theoperation part and the like. Thus, the user can further fine-tune theexposure time using the exposure time adjustment part 84 after theexposure time is automatically set, whereby a more detailed and accuratesynthesized image can be acquired.

The exposure time is presented on the display unit, so that the user canfurther fine-tune the exposure time using the exposure time adjustmentpart 84. For instance, a mode display region for displaying whichsynthesized image photography mode of the dynamic range wideningphotography mode and the resolution enhancing photography mode isselected, is provided on the display unit. Thus, the user can easilycheck the automatically selected synthesized image photography mode fromthe display screen. The user can also switch the synthesized imagephotography mode as necessary.

(Tone Conversion Part 86)

A synthesized image thus obtained by setting an appropriate synthesizedimage photography mode with the mode selecting part enables thesynthesized image photography mode to be grasped on the magnificationobservation apparatus side, and thus the information on the synthesizedimage photography mode can also be recorded when saving the file of thesynthesized image. When the tone conversion part 86 tone-converts thesynthesized image data, the information on the synthesized imagephotography mode is read out during reading operation to performappropriate image processing. Alternatively, information on the imagingcondition such as exposure time can be recorded to determine whether theobservation application of the synthesized image data is for the dynamicrange widening application or the resolution enhancing application basedon the information.

The observation purpose can be speculated and appropriate imageprocessing can be performed by performing image analysis of thesynthesized image or by analyzing the original image data configuringthe synthesized image even if such information is not present.

A case of converting the synthesized image data, which is a high toneimage, to a tone mapping image, which is a low tone image (LDR image),will be described as one example of performing tone conversion in thetone converting part. It should be noted that the present invention isnot limited to tone conversion of synthesized image data, and a processof converting a high tone image to a low tone image of narrower dynamicrange can be applied for the high tone image in general. For instance,tone mapping can be applied to a high tone image imaged and synthesizedin advance in another imaging device and a high tone image directlyimaged without using the original image.

(Drawing Setting Part 87)

In the present embodiment, the setting of drawing parameters relating toimage processing in converting the high tone image to the low tone imageand displaying or outputting the same can be appropriately andautomatically adjusted according to the type of high tone image.Specifically, the high tone image to be converted is automaticallydetermined by the drawing setting part 87 whether it is for the dynamicrange widening application or for the resolution enhancing application,and tone conversion is performed in the tone conversion part withhalation suppressing process set in the case of dynamic range wideningapplication and contrast enhancement process set in the case ofresolution enhancing application. The user can acquire a low tone imagesuited for a desired observation application without being conscious ofthe setting of drawing parameters by automatically performingappropriate image processing according to the type of distinguished hightone image.

When used in the dynamic range widening application, drawing isperformed with information on the region where saturation such asoverexposure or underexposure is occurring contained in the imageaccurately extracted, and luminance and color substantially the same asthose of the observed image before photographing needs to be maintainedfor other regions When used in the resolution enhancing application,fine patterns that can barely be distinguished in the observed imagebefore photographing need to be greatly enhanced, and thus the drawnimage may differ from the original image in color and the like.Therefore, the drawing setting part 87 sets the drawing parameterscorresponding to the type of image to be converted so that appropriatetone conversion corresponding to the type of image to be converted canbe carried out.

(Simple Mode)

FIGS. 17 and 18 show one example of a user interface screen of a toneconversion console screen for performing tone mapping in the magnifiedimage observation program. A tone conversion console screen 300 shown inFIG. 17 shows a simple mode in which the setting screen is simplified asan automatic setting screen example for automatically setting thedrawing parameters for tone conversion, and a tone conversion consolescreen 300B shown in FIG. 18 shows a detailed mode with the settingitems of the drawing parameters increased.

In the tone conversion console screens 300, 300B, a “simple mode” checkbox 366 is provided as a tone conversion setting switching part forswitching the setting part for tone conversion. That is, if the “simplemode” check box 366 is turned OFF in the tone conversion console screen300 of FIG. 17, the mode switches to the detailed mode shown in FIG. 18,and if the “simple mode” check box 366 is turned ON in the toneconversion console screen 300B of FIG. 18, the screen returns to thesimple mode screen shown in FIG. 17.

Sliders for manually adjusting the drawing parameters for tone mappingare provided in each of the tone conversion console screens 300 and300B. Specifically, the simple mode screen shown in FIG. 17 includes abrightness slider 312 and a texture enhancement slider 322. Currentlyset numerical values are displayed on the right side of each slider,which numerical values can be increased or decreased by adjusting theslider in the horizontal direction. The numerical values may bespecified through direct input.

(Detailed Mode)

In the detailed mode screen of the tone conversion console screen 300Bshown in FIG. 18, a contrast slider 332 and a color slider 342 areprovided in addition to the brightness slider 312 and the textureenhancement slider 322.

As shown in FIG. 19A to FIG. 22B, the quality of the low tone imagedisplayed on the display unit is changed by operating each slider.First, FIG. 19A shows an example where the brightness slider 312 is setlow and FIG. 19B shows an example where the brightness slider 312 is sethigh for the synthesized image of a ceramic imaged in the resolutionenhancing application of narrow dynamic range. The brightness slider 312can adjust the brightness of image by adjusting the y (gamma) value.FIG. 20A shows an example where the texture enhancement slider 322 isset low and FIG. 20B shows an example where the texture enhancementslider 322 is set high for the same sample. Thus, the textureenhancement slider 322 enhances fine patterns. In particular, in theresolution enhancing application, the details of the sample with smallcontrasting density that were difficult to observe can be observed byadjusting the drawing parameters.

FIG. 21A shows an example where the contrast slider 322 is set low andFIG. 21B shows an example where the contrast slider 322 is set high forthe synthesized image obtained by photographing and imaging a solderedcircuit of wide dynamic range serving as another sample in the dynamicrange widening photography mode. The contrast slider 332 roughly adjustsdarkness and brightness. FIG. 22A shows an example where the colorslider 342 is set low and FIG. 22B shows an example where the colorslider 342 is set high. The color slider 342 adjusts color or vividnessof color from monotone to color image. The user can manually adjust thedrawing parameters to acquire an appropriate tone mapping imagecorresponding to each observation purpose and can save the same asnecessary.

In the simple mode, the contrast and the color, which are two remainingdrawing parameters, can be automatically adjusted from, e.g., anestimation result of the brightness and texture enhancement degree setby the sliders, the drawing parameters after photographing, and thelike. Such estimation results are reflected on the value of each sliderwhen switched from the simple mode to the detailed mode.

Various buttons for reading out a file and saving the file are providedat a lower portion of each of the tone conversion console screens 300and 300B of FIGS. 17 and 18. In this example, an “HDR read” button 360for reading an HDR image, or a synthesized image, a “JEPG read” button362 for reading a tone mapping image including a synthesized image data,and a “JPEG save” button 364 for saving the tone mapping image includingthe synthesized image data are provided.

(Automatic tone conversion procedure of high tone image)

A procedure for tone-converting a high tone image to a low tone imagewill be described with reference to the flowchart of FIG. 23. In stepS2401, an HDR image is acquired, and in step S2402, the HDR image isonce converted to a gray image. In step S2403, the gray image is furtherconverted to a logarithmic image. In step S2404, the logarithmic imageis separated into a texture image and a sketch image. Specifically, thefrequency component of the luminance signal contained in the logarithmicimage is separated into a texture component relating to a fine patternand a component indicating the entire sketch using a frequency filterand the like. The texture image has features of an image extending overa wide dynamic range such as the image imaged in the dynamic rangewidening photography mode such as the HDR image. The sketch image hasfeatures of a fine pattern portion obtained in the image imaged in theresolution enhancing photography mode such as ceramic. Furthermore,holding or tone changing process on the texture image is performed instep S2405-1 for the texture image, and the tone changing process isperformed in step S2405-2 for the sketch image. As a result, microscopicconcave-convex portions are relatively enhanced when the tone changingprocess is performed in step S2405-1.

In step S2405-2, the dynamic range of the sketch image is compressed,whereby the brightness of the dark portion becomes relatively larger.The drawing parameter in step S2405-2 becomes a compression rate of thedynamic range of the luminance of the sketch image, and corresponds to“contrast” in the examples of FIGS. 17 and 18. In this regard, contrastrepresents the compression parameter of the dynamic range.

In step S2406, weighting addition is performed on each image and theseparated images are synthesized, whereby the gray image is outputted.The weighting referred to herein is distributed according to whether ornot to enhance the texture, how much pixels to saturate, and the like.The drawing parameter in step S2406 corresponds to “texture enhancementdegree” in the examples of FIGS. 17 and 18.

The gray image is converted to color image in step S2407. Furthermore,in step S2408, y correction and color saturation correction areperformed on the obtained color image to obtain an output image. The ycorrection corresponds to “brightness” in the examples of FIGS. 17 and18, and the color saturation correction corresponds to “color”.

Compression is efficiently performed since the component is separatedinto the texture component and the sketch component, and different imageprocessing is performed on each component. That is, with respect to thehigh tone image of dynamic range widening application, a phenomenon inwhich even the fine concave-convex components get compressed therebylosing the concave-convex feeling is effectively prevented whencompressing the luminance distribution. Thus, the drawing setting partspeculates whether the user is using the tone conversion function forthe purpose of dynamic range widening application or resolutionenhancing application, and appropriately sets the drawing parametersfrom the observed state before photographing, analysis results duringphotographing of the original image, characteristics of the high toneimage, and the like.

(Manual Adjustment of Drawing Parameters)

The user can manually set the drawing parameters independent of theautomatic setting. Alternatively, the user can further fine-tune thedrawing parameters to optimum drawing parameters after beingautomatically set by the tone conversion part. The tone conversionconsole screen 300B of FIG. 18 can be used for fine tuning, as describedabove.

(Setting Procedure of Drawing Parameters)

The setting procedures for a specific drawing parameter will now bedescribed with reference to the flowchart of FIG. 24. First, in stepS2501, the type of image to be converted is determined, and the texturegain a of the drawing parameter is set based thereon. Specifically, thetexture enhancement degree (texture gain a) of the drawing parameters isdetermined based on at least one of usage purpose of the user determinedwhen photographing the original image or the dynamic range of the hightone image. The texture gain a is set larger the smaller the dynamicrange is. The texture gain a is determined from the following equation.Texture gain a=k−w*log (DR)(k, w: constants, DR: dynamic range of high tone image)

The brightness of the image is then determined. The brightness can becontrolled with the contrast and the brightness (y) of the drawingparameters. First, in step S2502, the contrast is determined based onthe dynamic range of the high tone image. If the dynamic range is large,the contrast is made large. The weighted average of the pixel valueweighted so that the weight of the saturated region becomes small isobtained from the observed image before photographing. The weightedpixel value average where the image performed with the process up to theweighting addition of step S2406 is weighted with the weight is obtainedbased on the gain a and the contrast. In step S2503, y is determined sothat the two weighted pixel value averages become close values, that is,the change in brightness between the observed image before photographingand the display image becomes small.

Lastly, the color saturation correction parameter is obtained based onthe y and the contrast is obtained in step S2504. The color (colorsaturation correction parameter) is set large since a whitish imagewithout color is obtained as the value of y in y conversion becomeslarger.

Such adjustments can be performed automatically or can be performed bythe user. In this case, the operation of the plurality of sliders ofFIG. 18 corresponding to each parameter can be adjusted while beinglinked to each other. For instance, the slider is cooperatively operatedso that the color becomes larger the larger the y is. In the example ofFIG. 18, adjustment is automatically performed so that the brightnessslider 312 and the contrast slider 332 become low since the weight ofresolution enhancing application becomes higher as the textureenhancement slider 322 is operated to take a higher value. On the otherhand, since the weight of dynamic range widening application becomeshigher as the brightness slider 312 takes a higher value, and thus eachslider is cooperatively operated and moved so that the textureenhancement slider 322 becomes small and the contrast slider 332 becomeslarge. In addition, a switching part for turning ON/OFF the slidercooperative operation function may be provided. The drawing parameterssuch as compression rate and the like can be automatically set accordingto the dynamic range of the image to be converted or the observationpurpose.

(Setting of weighted Drawing Parameters)

Furthermore, in addition to the configuration of alternatively settingin the drawing setting part 87 the drawing parameters specialized foreither the dynamic range widening application or the resolutionenhancing application, a configuration of setting the drawing parametersweighted in view of the weight of each application can be adopted. Forinstance, the observation application is analyzed in the drawing settingpart 87, the weighting coefficient for each of the dynamic rangewidening application and the resolution enhancing application iscalculated, and tone conversion is performed in the tone conversion partaccording to the drawing parameters set accordingly. Thus, the weightingdrawing parameters taking the balance between the dynamic range wideningapplication and the resolution enhancing application into considerationcan be set without fixing the drawing parameters to either application,whereby more flexible image display and observation can be performed. Ifthe drawing setting part makes a false determination when automaticallydetermining the observation application, the display result reflectingthe other observation application to a certain extent can be expecteddue to the weighting of the drawing parameters.

The weight of both the observation applications is determined and thedrawing parameters corresponding thereto are set to calculate theweighting in the drawing setting part. Specifically, the setting of anappropriate drawing parameter is performed according to how high theweight of halation removal is, or how high the weight of contrastenhancement is. For instance, in the dynamic range widening application,the fine texture enhancement is moderately set and setting is made in adirection of suppressing change in brightness of the non-halationportion to enhance the weight of halation removal. On the other hand, inthe resolution enhancing application, the fine texture is largelyenhanced, and setting is made in a direction of enhancing the contrasteven if the image quality or the image tone is slightly changed.According to the directivity of the conditions, the drawing parameterssuch as the texture enhancement degree can be changed according to theweight of both the observation applications. The criterion for judgingthe weight of each observation application includes being based on theHDR image, setting of imaging condition during imaging and the like.

(Judgmental Criterion Based on HDR Image)

In the judgment using the HDR image, the ratio between the minimum pixelvalue and the maximum pixel value of the HDR image, that is, the dynamicrange of the HDR image becomes the judgmental criterion. In this method,the texture enhancement degree is moderately set if the dynamic range islarge since the ratio of halation removal application increases, and thetexture enhancement degree is set large if the dynamic range is small.The luminance distribution of the HDR image also can be used. If theluminance variance is large and the distribution of luminance extendsover a wide range, weight of halation removal is estimated to be high.

(Judgmental Criterion Based on Imaging condition)

In the judgment using imaging condition, the exposure time setting of acamera or an imaging part can be used. That is, the dynamic range of thescene is estimated by performing temporary photographing before the HDRimage photographing. The range of the exposure time used in the imagingof the HDR image reflects the dynamic range of the scene. That is, theweight of halation removal is high when the range of exposure time iswide, and the weight of contrast enhancement is high when the range ofexposure time is narrow or photographing is performed in a singleexposure time.

Therefore, the drawing parameters are set in view of the significance ofthe observation application on the dynamic range widening application orthe resolution enhancing application, and tone conversion is performedin the tone conversion part, so that balanced observation complying withthe observation purpose of the user can be carried out.

(High Tone Data-Attached Display File)

According to the present embodiment, the file format for saving thesynthesized image data may be the saving format in the form of attachingthe synthesized image to the tone-converted low tone image. The low toneimage data file may be converted to a versatile image format with a userregion such as JPEG and TIFF, so that the low tone image data portioncan be displayed in the general image display program. Through the useof a dedicated high tone image file creating program, the synthesizedimage data, which is the original data of the low tone image, can behandled, and the synthesized image data can be distributed whileenhancing the readability of the file.

(Saving File Format)

The HDR image data has tones greatly exceeding the eight-bit tone, andthus the original data will be damaged if saved in the same format asthe normal eight-bit image. Thus, the data is generally saved in theformat performed with sixteen bit, thirty-two bit floating point andother special coding. However, the image display software correspondingto such a special file format is not widely used, and most users cannotbrowse the HDR image on the normally using image display software. Inthe magnification observation apparatus according to the presentembodiment, the display image is saved in the normal image format ofJPEG and TIFF, and the HDR image is saved as the metadata. The user canthen use the display image with the normally using image displaysoftware.

(Display of High Tone Image)

The brightness, the texture enhancement degree, the color, and the likecan be changed by changing the drawing parameters. An example where thedrawing parameters are changed is shown in the image of FIGS. 25 and 26.The images shown in such figures are both display images created fromthe same HDR image. It is apparent that fine textures are enhanced inFIG. 26. Through the use of a dedicated image editing program capable ofchanging the drawing parameters, the drawing parameters of the HDR imagecan be changed by reading the metadata. Thus, the user can edit thephotographed HDR image in his/her computer to create various displayimages. Such texture enhancement can be applied to the general eight-bitimage, but fine gradation and the like may not be represented sincethere is a limit in the tone as opposed to the HDR image. Since the HDRimage has abundant tones, even fine gradation can be beautifullyrepresented.

The drawing parameters at the time of tone conversion of the low toneimage can be recorded in the high tone data-attached display file.Through the use of high tone image file creating program, the drawingparameters used in creating the low tone image can be checked. The userthus does not have to store the drawing parameters by taking notes forevery file, and there is obtained a high tone data-attached display filethat is extremely easy to use having the information managed in anintegrated fashion by the file.

Furthermore, the low tone image newly tone-converted with a differentdrawing parameter is updated to replace the low tone image data of thehigh tone data-attached display file. Thus, the readability of the lowtone image can be enhanced and retuning of the low tone image alsobecomes possible by providing the high tone image data serving as thebasis for generating the display low tone image data to the display lowtone image data.

(Structure of High Tone Data-Attached Display File)

FIG. 27 shows the structure of high tone data-attached display file.This figure is corresponded to the structure of a general image filesuch as JPEG and TIFF and includes a header region and a data region.The longitudinal and horizontal widths of the image and the like aregenerally stored in the header region. A user setting region to whichthe user can write is normally provided in the header region. A flagindicating the existence of high tone data is inserted to that region.The low tone image is saved as eight-bit data complying with JPEG, TIFF,and the like in the data region. A high tone image region is provided atthe tail of the low tone image data as metadata, and the header and thedata of the high tone image are recorded therein. Information such asthe file size of the synthesized image data is saved in the headerregion of the high tone image, and the actual data of the synthesizedimage is saved in the data region of the high tone image. The knownthirty-two bit floating point TIFF, log Yuv Open EXR, and the like areappropriately used in the saving format of the actual data. Theuncompressed or the lossless compression format can be adopted for thedata size. Furthermore, a drawing parameter holding region may beprovided in the high tone image region for recording the drawingparameter at the time of tone conversion of the low tone image. Thedrawing parameter holding region may be provided in the user settingregion.

The high tone data-attached display file saves the low tone image datain a versatile image format such as JPEG and TIFF of eight-bit displaysystem when the synthesized image of sixteen bits is tone-converted to alow tone image of eight bits. Since the user setting region that can beset by the user exists in such an image file, information indicatingthat the synthesized image data is contained and the drawing parametersat the time of tone-converting from the synthesized image to the lowtone image can be recorded in that portion. In addition to the fileformat using the floating point and the like, the synthesized image datacan be compressed in data size. Existing methods can be appropriatelyadopted for the data compression method. When saving the high tone imagedata to the file, the display image data at the time of displaying themulti-tone data in the conventional eight-bit display system with anappropriate tone conversion method is saved in a general image formatsuch as JPEG and TIFF, and the multi-tone data and the data such asdisplay parameters for creating the display image data from themulti-tone data can be added to the tail of the file, and then saved.

(Extraction and Separation of Texture Component and Sketch Component)

The time necessary for opening the file can be reduced by extracting thetexture component when creating the high tone data-attached display fileand adding the extracted texture component to the file. Conventionally,when displaying the synthesized image on the monitor and the like, thetask of extracting the texture component from the synthesized image isnecessary to display the image with the texture enhanced, and thus theprocessing time becomes long. The calculation amount of the textureextraction can be reduced by separating and saving the texture componentin advance in time of file saving, whereby the process from opening thefile to displaying the texture processed image becomes high speed or lowload. In this case, the file size of the high tone data-attached displayfile increases due to the attachment of the texture data. The sketchcomponent extracted with the texture component from the synthesizedimage data is saved, and the sketch component is saved in high data sizecompression rate, so that increase in file size can be suppressed. Inparticular, since change in sketch component is generally small, thedegradation of image quality is small even if the data size compressionrate is increased. Thus, the file can be efficiently saved whileincreasing the data size compression rate, reducing the file size, andsuppressing degradation of image quality.

(Corresponding Pixel Link Function)

The pixels of the low tone image and the synthesized image are linkedand an arbitrary pixel on the low tone image is specified, therebyproviding a corresponding pixel link function for displaying the pixelvalue of the synthesized image at the relevant position. The low toneimage displayed on the display unit sometimes cannot be displayed withthe original color due to tone converting from the synthesized image tothe low tone image. The data of the synthesized image corresponding tothe point specified on the low tone image can be referenced and theinformation can be read for display by linking each pixel whilecorresponding the coordinate position of the low tone image and thesynthesized image. For instance, when the user clicks an arbitrary pointon the low tone image with the pointing device such as mouse from thescreen of FIGS. 19A and 19B and the like, the pixel value on thesynthesized image corresponding to the relevant position is read, andinformation on the pixel value or the coordinate position thereof aredisplayed on the display unit. Two arbitrary points can be specified onthe low tone image, and the chromaticity difference can be calculatedfrom the corresponding position of the synthesized image and displayed.Such corresponding pixel link function is realized by functioning theimage calculating unit 81 as the corresponding pixel link part. Theslight change and the like in color that cannot be distinguished fromthe screen of the display unit can be recognized by associating the HDRimage and the LDR image in such manner.

(Procedures for Creating and Saving the High Tone Data-Attached DisplayFile)

The procedures for creating and saving the high tone data-attacheddisplay file will now be described. First, the synthesized image data inwhich a plurality of original images is synthesized is built and savedas necessary. In building the data, the texture component is extractedfrom the synthesized image.

The drawing process for displaying the synthesized image as a low toneimage will now be described. The drawing parameter in this case isautomatically set in the high tone image file creating program.Alternatively, the user may set the drawing parameter or may fine tunethe automatically set value. The versatile image format such as JPEG andTIFF can be used as the file format of the drawing processed image data.Since the user setting region that can be used by the user exists in theimage file, the flag data indicating the presence of the high tone imagedata, coupled region address of the high tone image data, and the sizeof the high tone image data are written to the relevant region.Furthermore, the low tone image file coupled with the high tone imagedata is created in the specified region in the data region. The drawingparameters are also saved in the drawing parameter holding regionprovided in the high tone image region. The instruction of specificsaving is performed by pushing the “JPEG save” button 364 provided atthe lower part of the tone conversion console screens 300, 300B of FIGS.17 and 18 as one form of the tone data saving part. The “save as”dialogue is displayed when the “JPEG save” button 364 is pushed, so thatthe user can specify the desired file name and specify the savingformat. The TIFF may be selected other than the JPEG for the fileformat. When saving only the synthesized image, thirty-two bit floatingpoint TIFF, log Yuv, Open EXR, and the like can be specified.Alternatively, JPEG, TIFF, BMP, and the like can be specified whendisplaying only the tone mapping image.

The high tone data-attached display file created in this manner candisplay the low tone image data portion with a general image displayprogram. The high tone image data-attached to the end of the high tonedata-attached display file can be displayed by using a dedicated hightone image file creating program. When the “JPEG read” button 362 ispushed from the tone conversion console screens 300, 300B of FIGS. 17and 18, the file select dialogue is displayed, so that the user canspecify the desired file. When opening the high tone data-attacheddisplay file with the high tone image file creating program, thecoupling flag of the high tone image data is first checked. In the caseof an image file coupled with the high tone image data, the high toneimage data is read, the drawing parameters saved in the drawingparameter holding region are read, and the drawing process is performedaccording to the drawing parameters, and then displayed.

The user can reset the drawing parameters to change the low tone imagein this state, where the old low tone image can be replaced and saved asthe changed new low tone image during file saving. When updating the lowtone image of the high tone data-attached display file, the drawingparameters of the drawing parameter holding region can be updated whenupdating the low tone image of the high tone data-attached display file.The coupling of the high tone image data can be released to separate thesynthesized image and the low tone image, or the separated synthesizedimage and the low tone image can be coupled.

Since the file format of the high tone image of the conventional HDR andthe like is not versatile, it cannot be displayed in most of the normalimage display programs, but it is suitable for distribution of high toneimage as at least the low tone image can be easily displayed with therelevant method and the content of the data can be understood. The hightone image data can be handled through the use of the high tone imagefile creating program, and both versatility and specialty of the filecan be satisfied. In particular, the low tone image in which the drawingparameters are adjusted can be replaced using the high tone image filecreating program. The drawback in that the file size of the high tonedata-attached display file becomes large by adding the high tone imagedata of large file size can be suppressed by adding the data whilecompressing the data size of the synthesized image data.

(Acquisition, Display, and Saving of Synthesized Image)

The flow of acquisition and display, and data saving of the synthesizedimage using the magnification observation apparatus according to thepresent embodiment is shown in FIG. 28. As shown in the figure, aplurality of eight bit images photographed while appropriately changingthe exposure time are synthesized, thereby generating the HDR image,which is the high tone image. The HDR image has a very wide range oftones, and the ratio of the maximum contrast and the minimum contrast isabout 10000:1. The HDR image is recorded using the floating point. Whendisplaying the HDR image on the display unit 52 such as a monitor, toneconversion (tone mapping) to the tone mapping image data of eight bits(265 tones) and the like that can be displayed on the display unit 52 isperformed. The high tone HDR image thus can be displayed as the eightbit color image even on the display unit having limited dynamic range.The tone mapping image greatly changes due to the drawing parametersetting in time of tone conversion process.

(Embedded Image File Saving)

The HDR image data and the tone mapping image data are respectivelysaved in a predetermined format, but the tone mapping image data can besaved with the HDR image embedded therein. The embedded image filerecords tone mapping image data at the head and the HDR image data atthe end. The HDR image data is handled as metadata. This file may be aversatile image format, and the tone mapping image data can be displayedwith the general image display program. Through the use of the dedicatedsoftware capable of handling the tone mapping image data, the tonemapping image data can be newly generated from the HDR image, and thetone mapping image data can be updated. Thus, use as the HDR image canbe achieved while maintaining the handiness of the image. In particular,the drawing parameters can be set and replaced so that the image of userpreference is obtained.

The magnification observation apparatus, the high tone image filecreating method, and the high tone image file creating method of thepresent invention generate and enable browsing of the HDR image usingmicroscope, reflective type or transmissive type digital microscope, anddigital camera.

1. A magnification observation apparatus capable of displaying an imageobtained by imaging a sample to be imaged according to a set imagingcondition, the magnification observation apparatus comprising: animaging condition setting part that sets at least an exposure time asthe imaging condition when imaging an original image in an imaging unit;the imaging unit that images the original image having a predetermineddynamic range, which is a ratio between minimum luminance and maximumluminance, according to the imaging condition set in the imagingcondition setting part with respect to an observation position of thesample; a synthesized image generating part that generates synthesizedimage data that is higher in tone than a tone width of the originalimage by synthesizing a plurality of original images imaged underdifferent imaging conditions at the same observation position of thesample; and a mode selecting part that selects, as a synthesized imagephotography mode for acquiring the synthesized image from the pluralityof original images imaged in the imaging unit, either a dynamic rangewidening photography mode for generating the synthesized image having adynamic range wider than that of the original image, and a resolutionenhancing photography mode for enhancing a luminance resolution from theoriginal image in a dynamic range narrower than that of the originalimage.
 2. The magnification observation apparatus according to claim 1,wherein the mode selecting part is configured to automatically select anappropriate synthesized image photography mode based on image analysiswith respect to the acquired image or the imaging condition.
 3. Themagnification observation apparatus according to claim 2, furthercomprising a display unit that displays the images imaged by the imagingunit, wherein the mode selecting part selects the dynamic range wideningphotography mode when a saturating portion exists, and the resolutionenhancing photography mode when the saturating portion does not existbased on a determination result as to whether or not the saturatingportion exists in the image displayed on the display unit whenautomatically selecting the synthesized image photography mode.
 4. Themagnification observation apparatus according to claim 2, wherein theexposure time suited for each synthesized image photography mode is setin the imaging condition setting part to image the plurality of originalimages while changing the imaging condition according to thedetermination result in the mode selecting part.
 5. The magnificationobservation apparatus according to claim 2, wherein when automaticallyselecting the synthesized image photography mode, the mode selectingpart selects the dynamic range widening photography mode when thesaturating portion exists and the resolution enhancing photography modewhen the saturating portion does not exist based on a determinationresult as to whether or not the saturating portion exists in anytemporary image, which is less in number than the number of originalimages to be imaged necessary for generating the synthesized image,imaged in a temporary imaging condition set in advance.
 6. Themagnification observation apparatus according to claim 5, wherein whenthe dynamic range widening photography mode is selected in the modeselecting part, a range for changing the exposure time from a range inwhich the exposure time is reduced to an extent where overexposure doesnot occur in all the regions of the temporary image to a range in whichthe exposure time is changed to an extent where underexposure does notoccur in all the regions of the temporary image is set in the imagingcondition setting part as a range for changing the exposure time toacquire the plurality of original images, and when the resolutionenhancing photography mode is selected, a temporary image without asaturated region and having the widest luminance distribution range isextracted from a plurality of temporary images, and a range for changingthe exposure time is set in the imaging condition setting part based onthe exposure time at which the extracted temporary image is imaged. 7.The magnification observation apparatus according to claim 2, whereinwhen automatically selecting the synthesized image photography mode, themode selecting part determines whether or not a dynamic range of atemporary synthesized image is larger than a dynamic range of theimaging unit based on a temporary synthesized image obtained bysynthesizing the temporary images, which are less in number than theoriginal images to be imaged necessary for generating the synthesizedimage, imaged under the temporary imaging condition set in advance, andselects the dynamic range widening photography mode when the dynamicrange of the temporary synthesized image is larger and the resolutionenhancing photography mode when the dynamic range of the temporarysynthesized image is smaller, the range for changing the exposure timebeing set in the imaging condition setting part according to theselected synthesized image photography mode.
 8. The magnificationobservation apparatus according to claim 7, wherein when the resolutionenhancing photography mode is selected in the mode selecting part, thesynthesized image generating part generates synthesized image data byimaging a plurality of original images in the imaging unit with theexposure time fixed to a constant value and obtaining a frame averagevalue.
 9. The magnification observation apparatus according to claim 1,further comprising an exposure time adjustment part that further adjuststhe exposure time of the original image set by the imaging conditionsetting part to a desired value.
 10. The magnification observationapparatus according to claim 2, further comprising a tone conversionpart that converts synthesized image data generated by the synthesizedimage generating part to low tone image data having a tone width capableof being displayed on the display unit.
 11. The magnificationobservation apparatus according to claim 2, further comprising a drawingsetting part that sets a drawing parameter relating to image processingfor tone-converting the synthesized image to low tone image data in thetone conversion part according to a feature of an image to be converteddistinguished from the image to be converted, wherein image processingis performed by the tone conversion part according to the drawingparameter set in the drawing setting part and display is made on thedisplay unit.
 12. A magnification observation apparatus capable ofdisplaying an image obtained by imaging a sample to be imaged accordingto a set imaging condition, the magnification observation apparatuscomprising: an imaging condition setting part that sets at least anexposure time as the imaging condition when imaging an original image inan imaging unit; the imaging unit that images the original image havinga predetermined dynamic range, which is a ratio between minimumluminance and maximum luminance, according to the imaging condition setin the imaging condition setting part with respect to an observationposition of the sample; a synthesized image generating part thatgenerates synthesized image data that is higher in tone than a tonewidth of the original image by synthesizing a plurality of originalimages imaged under different imaging conditions at the same observationposition of the sample, wherein a weighted imaging condition obtained byweighting and combining the imaging conditions with respect to eachapplication is set in the imaging condition setting part depending onwhether the application is a dynamic range widening photographyapplication for generating the synthesized image having a dynamic rangewider than that of the original image or a resolution enhancingphotography application for enhancing a luminance resolution from theoriginal image in a dynamic range narrower than that of the originalimage in order to acquire the synthesized image from the plurality oforiginal images imaged in the imaging unit.
 13. A magnificationobservation apparatus capable of displaying an image obtained by imaginga sample to be imaged according to a set imaging condition, themagnification observation apparatus comprising: an imaging conditionsetting part that sets at least an exposure time as the imagingcondition when imaging an original image in an imaging unit; the imagingunit that images the original image having a predetermined dynamicrange, which is a ratio between minimum luminance and maximum luminance,according to the imaging condition set in the imaging condition settingpart with respect to an observation position of the sample; asynthesized image generating part that generates synthesized image datathat is higher in tone than a tone width of the original image bysynthesizing a plurality of original images imaged under differentimaging conditions at the same observation position of the sample,wherein synthesized image photography mode for acquiring the synthesizedimage from the plurality of original images imaged in the imaging unitincludes a resolution enhancing photography mode for enhancing aluminance resolution from that of the original image in a dynamic rangenarrower than the original image.
 14. A method for photographing amagnified image comprising the steps of: setting an imaging conditionincluding an exposure time of an imaging unit; imaging a plurality oforiginal images in the imaging unit according to the imaging condition;synthesizing the plurality of original images and generating synthesizedimage data, wherein the step of setting the imaging condition includes:determining whether or not a saturated region exists in an imagedisplayed on a display unit, and switching to a dynamic range wideningphotography mode when the saturated region exists and to a resolutionenhancing photography mode when the saturated region does not exist; andsetting the exposure time such that amount of change in the exposuretime set in the resolution enhancing photography mode becomes smallerthan the amount of change set in the dynamic range widening photographymode when setting the exposure time as the imaging conditioncorresponding to the selected synthesized image photography mode.
 15. Amethod for photographing a magnified image comprising the steps of:setting an imaging condition including an exposure time of an imagingunit; imaging a plurality of original images in the imaging unitaccording to the imaging condition; synthesizing the plurality oforiginal images and generating synthesized image data, wherein the stepof setting the imaging condition includes: prior to imaging theplurality of original images in the imaging unit, determining whether ornot a saturated region exists in a plurality of temporary images imagedunder a temporary imaging condition set in advance, and switching to adynamic range widening photography mode when the saturated region existsand to a resolution enhancing photography mode when the saturated regiondoes not exist; and setting the exposure time such that the amount ofchange in the exposure time set in the resolution enhancing photographymode becomes smaller than the amount of change set in the dynamic rangewidening photography mode when setting the exposure time as the imagingcondition corresponding to the selected synthesized image photographymode.
 16. A method for photographing a magnified image comprising thesteps of: imaging a plurality of temporary images in an imaging unitaccording to a plurality of temporary imaging conditions set in advance;generating a temporary synthesized image by synthesizing the pluralityof temporary images; determining whether or not a dynamic range, whichis a ratio between minimum luminance and maximum luminance, of thetemporary synthesized image is larger than a dynamic range of theimaging unit based on the temporary synthesized image; selecting adynamic range widening photography mode when the dynamic range of thetemporary synthesized image is larger and a resolution enhancingphotography mode when the dynamic range of the temporary synthesizedimage is smaller; setting an exposure time to image original imagesgreater in number than the temporary images according to the selectedsynthesized image photography mode; imaging the plurality of originalimages in the imaging unit according to the setting of the exposuretime; generating synthesized image data by synthesizing the plurality oforiginal images; and converting the synthesized image to low tone imagedata having a tone width suited for display, and displaying the image ona display unit.